Papers with merged model
LoRA Soups: Merging LoRAs for Practical Skill Composition Tasks (2025.coling-industry)
Copied to clipboard
| Challenge: | Low-Rank Adaptation (LoRA) is a popular technique for parameter-efficient fine-tuning of Large Language Models. |
| Approach: | They propose to combine LoRA modules to achieve skill composition . they propose to use concatenation of LoRAs to optimize weights for different LoRA training . |
| Outcome: | The proposed model outperforms existing models and data- merging techniques on math-word problems and domain-specialized corpora. |
Merger-as-a-Stealer: Stealing Targeted PII from Aligned LLMs with Model Merging (2025.emnlp-main)
Copied to clipboard
| Challenge: | Model merging is a promising approach for updating large language models . but unmonitored mergers can introduce significant security vulnerabilities . |
| Approach: | They propose a model merging attack surface where a malicious merger can extract PII from an aligned model with model merg. |
| Outcome: | The proposed framework can extract PII from an aligned model with model merging. |
DivMerge: A divergence-based model merging method for multi-tasking (2026.eacl-long)
Copied to clipboard
| Challenge: | Existing methods for multitask learning struggle with interference between tasks, especially as the number of tasks grows. |
| Approach: | They propose a reference-free method that minimizes the divergence between models' outputs and those of the merged model, automatically balancing task importance. |
| Outcome: | The proposed method outperforms existing methods on classification and generative tasks and remains robust when scaling to more tasks. |
Be Cautious When Merging Unfamiliar LLMs: A Phishing Model Capable of Stealing Privacy (2025.findings-acl)
Copied to clipboard
| Challenge: | Model merging is a widespread technology in large language models that integrates multiple task-specific LLMs into a unified one. |
| Approach: | They propose a model merging approach that trains a phishing model capable of stealing privacy using a privacy phish instruction dataset. |
| Outcome: | The proposed model cloaking method mimics a specialized capability to conceal attack intent, luring users into merging the phishing model. |
To See a World in a Spark of Neuron: Disentangling Multi-Task Interference for Training-Free Model Merging (2025.emnlp-main)
Copied to clipboard
Zitao Fang, Guodong Du, Shuyang Yu, Yifei Guo, Yiwei Zhang, Yiyao Cao, Jing Li, Ho-Kin Tang, Sim Kuan Goh
| Challenge: | Existing approaches to model merging ignore the fundamental roles of neurons, connectivity and activation. |
| Approach: | They propose a framework that relies on neuronal mechanisms to mitigate task interference . they decomposed task-specific representations into two complementary subspaces . their results offer new insights into mitigating task interference and improving knowledge fusion . |
| Outcome: | The proposed framework reduces task interference within neurons and improves knowledge fusion. |